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FOREWORD 

by James C. Kielsmeier 

Service-learning, as a teaching and learning methodology, is expanding around the country. 
The need both to document the impact of service-learning on teaching and learning and identify 
ways to improve service-learning practice is critical to ensure long-term success. Moreover, as the 
authors make clear, researchers by themselves cannot begin to ask and answer the many 
questions that current service-learning practice raises. Teachers are positioned not only to 
question the methods and results of service-learning, but they also are equipped to help improve 
practice by beginning to answer their own questions. 

In 1995 in Philadelphia, with the support of the Corporation for National Service and the 
W.K. Kellogg Foundation, the National Youth Leadership Council spearheaded the first annual 
Service-Learning Research Seminar (preceding the National Service-Learning Conference) to 
focus attention on the vital role of research in the service-learning field. Participants included 
researchers and evaluators, service-learning practitioners, school administrators, and policy 
planners. The Action Research and Evaluation Guidebook for Teachers is a compilation of the 
presentations made at the third annual Service-Learning Research Seminar in Orlando in 1997. 
This unique guidebook provides service-learning professionals with a useful tool for designing 
and conducting their own action research. This guidebook is one in a series entitled Making the 
Case for Service- Learningy^ developed by the National Youth Leadership Council. 

The National Youth Leadership Council extends heartfelt thanks to the Corporation for 
National Service, Learn and Serve America, and the South Carolina Department of Education 
for their commitment and support of the Service-Learning Research Seminar and the 
development of this guidebook. Special thanks are due to Irving Buchen, Kathy Gibson Carter, 
Marty Duckenfield, Jim Ennis, Jay Smink, and Alan Waterman for their hard work and 
encouragement in the development, production, and dissemination of this guidebook. The 
National Youth Leadership Council would also like to thank the authors who contributed 
articles to this guidebook. 



^See the list of piiblicntions on page 30, under Resource Section. 
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WHY EVALUATE? 

Perhaps the idea of research and evaluation are concepts you thought you left behind when 
you entered the classroom. After all, isn’t that the role for researchers at universities and 
think tanks? What does it have to do with you and your role as a classroom teacher? And, as 
you work to develop a service-learning program, what does research and evaluation have to 
do with your work to integrate service into your curriculum? 



As a part of the education community, you have an extremely important role to play in the 
advancement of this teaching methodology. Let us look at why this research and evaluation 
is important, not only to the educational community, but also to you in your role as a 
service-learning teacher, and why it is important for you to be involved in the process. 
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INTRODUCTION 

by Jay Smink 

The Action Research and Evaluation Guidebook for Teachers has been prepared to provide 
teachers with a simple tool — one that can assist them in playing a vital role in the research and 
evaluation of service-learning that must take place if this teaching method is going to become a 
part of valid educational reform. 



Research and Evaluation Purpose 

There are several reasons why research and evaluation are necessary functions in the 
advancement of service-learning. 

1. Documenting results of service-learning programs. 

Service-learning is a relatively new teaching strategy in the education community and is even 
more foreign to the corporate and community leaders that are part of many educational 
partnerships. Successful stories and statistics illustrating the effectiveness of service-learning as a 
teaching methodology need to be shared with many different publics in order to g^in widespread 
support. 

2. Demonstrating quality service-learning programs. 

The education community needs to define and demonstrate quality service-learning 
programs. Program components that are successful and most effective need to be identified, as 
do those that are less successful. 

3. Ongoing monitoring of service-learning programs. 

Another purpose for having sound research and evaluation is to provide an opportunity for 
the program practitioners to monitor the project. This gives them the feedback they need to 
make any necessary program modifications. Service-learning programs can make improvements 
based on these evaluation results. 

Evaluation efforts should be a required and integral part of every service-learning program. A 
well-planned evaluation provides information that will show the impact of a program on the 
students, teachers, and other participants; what components make a quality program; and 
whether or not a program is operating as it was designed to operate. With information such as 
this, service-learning can both improve and grow. 
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Need for Teacher-Based Action Research and Evaluation 

Basic Research 

Research about basic learning strategies, longitudinal studies, or administrative studies are a 
few examples of research and evaluation studies that are typically conducted by college or 
university researchers. There will always be a need for these more formal and highly structured 
research and evaluation approaches. 

Action Research 

The current most critical need in the field of service-learning, however, is to capture the 
evaluation information available from hundreds of local school and community programs; this is 
called action research. Teachers engaged in service-learning projects are best positioned to witness 
the effects of the program on the students and others. It just makes good sense to promote 
teachers as having the major responsibility for completing action research regarding their own 
programs. In addition, hiring professional evaluators is generally too costly to be practical. 

This guidebook gives classroom teachers an overview of action research and evaluation 
techniques. It will also help teachers begin to design and conduct quality evaluation programs 
within the framework of their existing projects and daily teaching assignments. 



Role of Classroom Teacher as Evaluator 

1. Review Objectives and Collect Data 

The classroom teacher acting as an evaluator must not only serve as the coordinator to 
implement the program; the teacher must also be the professional with the insights to review the 
program objectives and collect the data to test how well the objectives were met. This is not a 
difficult task, but it does demand integrity in the handling of data as it is collected, analyzed, 
and reported. 

2. Present Data to Public 

The one role that will be the easiest to perform by the teacher-evaluator is the public relations 
responsibility. Classroom teachers are normally very happy to disseminate the results of the 
service-learning program to the various publics that need to hear about the student outcomes 
and other gains from the services provided in the community. Who is in a better position to 
provide evidence of program effectiveness than the enthusiastic classroom teacher involved in the 
program? Furthermore, teachers are in the very best situation to talk to other teachers and 
professionals about their own results and are able to give direct assistance to others wanting to 
replicate a similar service-learning program. 
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3. Maintain Objectivity 

When a classroom teacher assumes the task of serving as the evaluator, perhaps the most 
important aspect of that task is to be sure an objective perspective can be maintained. That 
unbiased perspective begins with the development and implementation of the evaluation design, 
continues with the collection and analytical procedures to be used, and ends with the style and 
methods used to report the results. The teacher may need to have assistance to design some of 
the more technical evaluation designs, but generally those more elaborate designs are not 
necessary. Teachers should engage students, community partners, and other key players in 
evaluation design, both for their essential contributions to the process and as a check on the 
teacher’s unrecognized biases. The basic evaluation design and procedures outlined in this 
guidebook are adequate to demonstrate program impact and share those results with other 
practitioners. 



Classroom Teachers — the Best Evaluators 

Classroom teachers coordinating service-learning activities as part of their curriculum are 
truly in the best position to evaluate their own programs. Based on their program and teaching 
objectives, they will know what indicators to use to measure and how to collect the most 
appropriate information. They will be able to analyze the data and see the changes in student 
behaviors and attitudes. They will be able to witness the community involvement and the trends 
in student growth and other teacher involvement. And with true personal convictions, classroom 
teachers can offer the best evaluation results to other practitioners, policy makers, researchers, 
and community leaders. 



We have seen why teachers, with their position at the frontlines, can make such a needed 
contribution to servicedearning research and evaluation. Our next section shows why 
participation in this component also improves your teaching and the learning that will take 
place with your students. 
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CONNECT SERVICE WITH EVALUATION 
FOR GOOD LEARNING 

by Robert D. Shumer 

Service-learning and evaluation are joined at the hip. You can’t do one without the other. The 
Alliance for Service-Learning in Educational Reform (ASLER) principles of good practice suggest 
that all high quality service-learning programs include “formative and summative evaluation.” In 
fact, all good service-learning programs do more — they incorporate evaluative processes 
throughout the service and learning activities. Evaluation begets learning, and vice versa. 

Exemplary service-learning programs include several components that utilize evaluation 
processes and techniques to measure learning, impact, and program improvement. These require 
the learner and learning system to determine community needs and to assess community 
outcomes to monitor the effectiveness of the service itself With evaluation considered an 
external phenomenon for conventional learning situations, service-learning makes it integral to 
the educational methodology, requiring evaluative processes be applied in order to develop the 
instructional program. Service-learning includes evaluation in the very essence of its being, from 
the student, to the program, to the community served. In service-learning the student shares 
control of the evaluation with teachers, community members, and all those affected by the 
service activities. 

Given this connection, there are many reasons to include evaluation in exemplary service- 
learning, and there are at least eight concerns which drive the evaluation process (Shumer, 1997). 

1) Students perform community assessments to determine program emphases. 

2) Students evaluate the impact of service activities on the community. 

3) Students evaluate their own learning from the service activities. 

4) Instructors need to evaluate whether students are learning appropriate 
information/skills/concepts from the activity. 

5) Program managers determine whether the service-learning method is an effective 
instructional strategy. 

6) Community members determine the impact of the service on local improvement. 

7) School district and state standards are assessed in terms of student learning. 

8) Funders and others who financially support service-learning determine whether or not to 
continue their support based on established criteria of success. 
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Besides these evaluation needs, Patton (1990) describes six areas critical to any plan of study. 
Before creating a design, one should consider the following questions: 

1) Who is the information for and who will use the findings? 

2) What kinds of information are needed? 

3) How is the information to be used? For what purposes is evaluation being done? 

4) When is the information needed? 

5) What resources are available to conduct the evaluation? 

6) Given answers to the preceding questions, what methods are appropriate? 

Patton, 1990, p. 12 

While these questions help structure designs for program evaluations, they are pertinent for 
any assessment. Questions, in fact, are the driving force for most evaluations (Shumer & Berkas, 
1992). They frame the intent for students, for teachers, and for all participants m the service- 
learning process. 

Melchior and Bailis (1996), two evaluators of national service-learning programs from 
Brandeis University, recommend some general guidelines for doing evaluation. They suggest 
tht.re are four critical starting points for teachers who want to evaluate their programs. 1) 
defining goals, 2) clarifying expectations, 3) looking at information already collected, and 4) 
documenting what is done. By following these points, teachers are able to connect program 
purposes with student activities that produce naturally occurring sources of data. Analysis of this 
data by both students and teachers produces the information needed to demonstrate what 
students learn and what has been accomplished by the program. 

Evaluation is a necessary component of any service-learning activity or program. It does not 
sit reflectively at the end of the program; it is a daily, weekly, monthly, and yearly process which 
improves the level of both service and learning. There are many reasons to do evaluation as part 
of service; the most important is to always determine what needs to be done, what needs to be 
known and learned to deliver the service, and what impact the service is having on the 
community, the participants, and the organization. Focusing on these elements assures that there 
is intentional learning that elevates the service from simple volunteerism to acts of citizenship 
and education. 



6 



12 



Making the Case for Service-Learning 



FRAMEWORK FOR 
ACTION RESEARCH 

It is always easier to have a step-by-step process. The model for action research found in this 
section will provide you with the framework you need to construct your research and 
evaluation plan. 

A MODEL OF ACTION RESEARCH 
FOR PRACTITIONERS 



by Irving H. Buchen 

Action research grows out of reflection. Students reflect on practice; so do teachers. The 
difference is that teachers and practitioners are motivated by professional curiosity and 
improvement. Why did this work? Why did this not work? The rationale for the first question is 
replication — to make sure it can work again and again, and maybe by knowing more, at a higher 
level. The second question proceeds obviously out of frustration and sometimes bewilderment. 
The expectation clearly was that it would be effective, but it wasn’t. Why? 

In either case, action research is the vehicle for facilitating practitioner development. Action 
research is thus a discovery process. It is inquiry learning. It is done by teachers, for teachers, and 
about teachers. It also may supplement or extend your project evaluation. Certainly, a grant 
provider would be delighted to have you turn in a completed research project as part of your 
evaluation. Happily, also, the research process is not terribly complicated. Indeed, it is essentially 
a five-step process. 

Action Research Process Model 

1) The Research Topic or Subject 

2) The Research Design 

3) Data Collection and Analysis 

4) Findings and Conclusions 

5) Communication and Dissemination 

Action Research & Evaluation Guidebook for Teachers 




Let us walk through and examine each one of these steps separately and in detail. 



1) Research Topic or Subject 

This initial stage may involve three operational activities: observation, discussion, and 
formulation. Generally, practitioners do not start off with a research project but with a research 
question or issue, generated by their own observation of their practice. This worked. This didn’t 
work. Why? Or this was supposed to have this impact or produce these results. But it didn’t. 
Why? 

If you are working with other teachers it is a good idea to discuss the issues that your 
observations have raised. Together, the dialogue can massage and move the issue closer to 
becoming a research project. That kind of dialogue can also identify some blind spots and 
variables that you want to take care of or acknowledge at this beginning stage. Finally, what 
emerges is that last activity, that of formulation. The problem you are investigating and hoping 
to solve is cast in final research form. 



2) Research Design 

If the first step was driven by "Why? ” this second step is driven by How? But the why 
drives the how. Each research project may have both generic as well as special and even unique 
ways of being studied. 

One of the first things you need to do is find out whether this research topic already has been 
done. This is accomplished by undertaking a literature review. Assuming you find that no one 
has done it yet, another value of the literature review of the field is that it will make you aware of 
research designs or methodologies used by others. 

Inevitably, you will need to establish baseline data. Whether you use pre- and post tests or 
school records, you need some basic and fixed points of comparison and contrast — some 
established point to measure change in the school and/or community. Such data insures that you 
are aligning apples and apples. In addition, every research design needs to spell out two factors: 
first, what outcomes you expect to have; second, what methods you plan to use to measure those 
outcomes. The methodology provides a natural transition to the next step. 



3) Data Collection and Analysis 

This is a critical step in the process because the credibility of the conclusions depends on the 
quality of the data collected. Various techniques can be used to gather data in the school and/oi 
community for both qualitative and quantitative outcomes: logs or diaries, interviews, 
questionnaires, video or audio tapes, portfolios, case studies, examination of GPA, attendance 
records, graduation rates, etc. 
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The analysis must be systematic; for example, you cant change data sets midstream. If you 
find the statistical analysis formidable, consult with a local college professor of statistics or 
someone in your administration. This is perhaps the most exciting step because here is where the 
conclusions begin to emerge. 

4) Findings and Conclusions 

You have to make sure that there are clear linkages between both the research question posed 
and the listed outcomes and conclusions you reach. In other words, they all must match. You 
also should link, if appropriate, your findings to those you found in the literature review. 

Because the inquiry process often begets other research opportunities, there should be a portion 
of your findings which spells out implications for further research. Finally, remember that 
practitioner research is done for the benefit of other practitioners; so take some care that the 
language in which you express your findings is user-friendly. Also, action research should be 
followed by action planning — how these results can be used to improve practice. 

5) Communication and Dissemination 

Research is not a private but a sharing enterprise. The results need to be reported in many 
different ways. Most immediately, they can be shared locally, in your school, in your district. You 
can volunteer to offer an in-service workshop. If you have a county or statewide electronic 
network, the research findings can be conveyed electronically. A particularly effective 
dissemination outlet is to make a presentation at a conference. Naturally, there is publication in 
newsletters, magazines, and journals. This is particularly gratifying because in effect you are 
paying back the help you received from others through the literature search as well as becoming 
part of the bibliography of your field. Finally, there is the ultimate conversion: introducing your 
students to and offering them the special opportunity of becoming researchers in a community 
of inquiry learners. 
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THE RESEARCH TOPIC 
OR SUBJECT 

The next section and those that follow focus on each stage of the action research framework. 
Thinking about some basic questions ahead of time is an important first step. 



EARLY QUESTIONS TO ASK 
IN PLANNING ACTION RESEARCH 



by Yolanda Yugar 

This section poses questions that will help you through the early phases of developing an 
action research plan. Knowing what you want to study and how you want to proceed is critical. 

If time is not taken in the beginning to consider these matters, you may run into unexpected 
obstacles and have to make major changes midstream. For example, you may realize that you 
have not collected the data you need or have collected much more than you actually require. 

Even worse, you may find that the key people who make decisions about your program are 
interested in outcomes other than those issues you chose to study. Considering certain crucial 
questions before you begin to formulate your plan of action will help you determine the whys, 
whats, and hows of your research and will save you time-consuming work in the long run. 

You want to begin by answering some very basic questions. This is by no means a complete 
list, but the questions suggested here will help in narrowing your focus so that the process doesn’t 
seem so overwhelming. Each answer may guide you down a different path or lead you to another 
set of questions. You do not need to be an expert in research to make these types of decisions. 
The first round of questions is related to deciding upon the research issues you will want to 
address. 

Who are the stakeholders, and more importantly what are they interested in knowing? School 
boards may want to see an increase in test scores or grades; parents may want to see that their 
children are excited about school, are having opportunities to explore career options, or are 
developing civic responsibility; and teachers may be interested in service-learning in relation to 
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special populations — those who are, for example, gifted, at risk, or mentally or physically 
challenged. Whoever the stakeholders are, knowing what they are interested in will help you 
focus on what research issues to study. 

The kind of outcomes you are looking for will also influence the research issues you will 
address. Are student GPAs improving? Are absenteeism or discipline referrals down? Are positive 
changes occurring in the community as a result of student service? Are teachers’ attitudes toward 
teaching improving? Student, community, and teacher outcomes are those most commonly 
assessed. The type of outcomes identified are often directly related to who the stakeholders are. 

How you decide upon the unit of analysis, or in other words, who or what you are 
evaluating, is also directly related to the type of outcomes, the stakeholders, and the research 
issues you will ultimately address. Who will be your target population? Will you study 
individuals (either students, teachers, or community members), the classroom, or the school 
district? Your unit of analysis will determine the methodology, design, and instrumentation of 
your research study. 

Deciding how you want to present your discoveries before you begin is useful. How do you 
want to display your findings? Will you include graphs, charts, or figures? For whom are you 
writing this report? Keeping your audience in mind and knowing how you want your final 
product to look will also help you select the data you will collect. 

Once you have decided upon the research issues you want to address, the next round of 
questions involves methodology, design, and instrumentation. These questions will help to guide 
the research process. 

1) How will you go about collecting your data, and when will you collect it? In technical 
terms, what is your methodology? 

2) Will your data be primarily quantitative (numbers), or qualitative (attitudes, beliefs, or 
opinions), or a combination of the two? 

3) What instruments, surveys, or questionnaires will you use to collect the data, and where 
will you get them? Will you use personal observations, small group discussions, or 
interviews with students, teachers, and community members? Will you use data that has 
already been collected, such as grades, attendance, or discipline referrals? 

4) What type of research design will you use: experimental with a control group; quasi- 
experimental, with existing classrooms; or case-studies, in-depth looks at a few individuals 
or classes? 
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Another suggestion that will help in answering these questions is to find out about existing 
research in the area you are considering for study. Knowing what has already been accomplished 
in the area can provide you with a model to follow as you consider your research. There are a 
number of ways to access existing research; the Internet, the National Service- Learning 
Clearinghouse, the ERIC Database, and your state’s Education Department or Learn and Serve 
America Coordinator are all places to start. There is no need to reinvent the wheel. Replication is 
flattering, and similar results will support previous work, while conflicting results chart new 
territory. 

Finally, it is important to note that no matter how simple or complex you decide your -study 
be — ^whether it will represent an individual, a classroom, or a state, or will seek to address 
one research issue or a series of issues — this same process of decision making should be followed. 
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THE RESEARCH DESIGN 

As you begin to look at issues to research and evaluate, perhaps this next section will stretch 
your thinking Creative and unique perspectives may provide the field with stronger results. 



NEW PARADIGMS FOR 
EVALUATING SERVICE-LEARNING 

by L. Richard Bradley 

After years of studying the impact of service-learning programs on elementary and secondary 
school students, consensus is still lacking about their real benefits. Numerous studies have 
suggested positive impacts in the areas of behavior (Benson, 1990), prevention of substance 
a use and dropout (Duckenfield and Swanson, 1992), gains in personal and social responsibility 
and improved academic performance (Conrad and Hedin, 1982, 1987, 1989), and increased 
problem-solving skills (Newmann and Rutter, 1983). However, establishing direct links between 

service-learning program objectives and program impacts has been difficult for a number of 
reasons: 



1) Ongoing confusion about what service-learning is and is not, resulting in confusion about 
what researchers should really be studying. 

2) Changes researchers would most like to study— changes in student attitudes and 
behaviors typicaily take more than one school year to appear, while most service- 
learning programs engage students for much shorter periods of time. 

3) Over reliance on quantitative methods of data collection and analysis, to the exclusion of 
other kinds of information that might help document actual program impacts. 

This suggests that a more inclusive approach to the assessment of service-learning is needed, 

one which follows the paradigm of nonlinear science and “listens to and is intimately engaged 
with the subject matter.” 

The word "assessment" comes from the Latin word, "assidere, ’’meaning "to sit beside. ” 

What would it look like for researchers to sit beside, to listen to,” the subject matter of service- 
arning. What are some of the questions this approach to evaluation might prompt them to ask? 
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More importantly, what additional information about program impacts might be obtained by 
taking this approach? 

For one thing, the new approach would invite researchers to pay closer attention to 
environmental factors surrounding the program, such as 

. conditions in the school and among students; 

• conditions in the community; and 

• teachers faithfulness to program intent. 

For another thing, teseatchem would also need to tecogniae that there ate really three levels of 
service-learning evaluation, each focusing on different questions and. thetefote, providing an 
interpretation of what happened. To fully undeBtand what happened in a particular program, 
information from all three levels is needed. 

At the most kasic kvel ate questions related to the kinds of information desired by 
stakeholders and funders who want to know 

. the number and ethnicity of students who participated in service activities, 

• the type and number of hours of service they performed, and 
. who the primary beneficiaries of the service were. 

This information is then used to make decisions about whether to continue funding a given 
program. 

At the intermediate level are questions related to deepening understanding of what happened 
in this program. Here, interest centers on 

. how teachers see service-learning methods connecting with what they teach, 

. the level of service-learning implementation-that is, the length and duration of student 
service-learning activities, 

. the kinds of reflective activities used and the amount of time spent in reflection, 

• the barriers teachers encountered and what they did to compensate, 

• the effectiveness of training and preparation, and 

. assessing the overall level of support for the program in the school and community. 
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This information, gathered by means of interviews and surveys, can give teachers and 
researchers more accurate information about why a particular program did or did not have the 
expected impact. This can be used to redesign future programs. 

At the deepest level are questions which focus on the impact the program has on participants 
and the community served. This involves using more quantitative methods — pre- and postservice 
assessments, use of comparison groups and archival data — to assess changes in student 
knowledge, skills, attitudes, and behaviors. Information collected at this level helps answer 
questions posed by administrators, parents, and teacher colleagues about the worth and value of 
service-learning programs. 

Service-learning is only a small part of what goes on in a typical school. Without carefully 
designed evaluations, incorporating the appropriate quantitative and qualitative methods, it will 
probably be viewed by teachers, parents, and administrators as tangential to the schools essential 
mission. “Listening to the environment” can help practitioners design and implement programs 
that put service-learning activities at the core of a schools academic mission. 
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DATA COLLECTION 

Service-learning provides a surfeit of riches; experiences galore and achievements in and out 
of the classroom are only a few of the available options to be examined by the evaluator. 

This section discusses how to purposefully comb this vast array for the data that will tell us 
what is really happening. 



INNOVATIVE MEASURES 

FOR EVALUATING SERVICE-LEARNING 



by Carol G. Weatherford 

Service-learning experiences, projects, and programs have meaning to those who plan and 
participate in them. In fact, it is the process of “meaning making” that is the power of the 
service-learning method. To evaluate service-learning in ways that ignore, overlook, or disregard 
meaning making, the very heart of service-learning, is to commit the same mistake as the new 
husband who cut the ends of the ham off before cooking it because his father always had cooked 
it that way, because his father always had cooked it that way. The new wife asked. Why? They 
all discovered the reason the grandfather had always cut the ends off the ham before cooking it 
was to make it fit his pan! 

In deciding what data to gather and how to gather it, remember the lesson learned from the 
ham: Don’t cut the ends off the ham before cooking it when you don’t need to. In other words, 
the questions you want to study should determine the kinds of data you need. The pan size from 
two generations ago should not dictate how you prepare the ham. It is in this sense that this 
chapter addresses its title, Innovative Measures for Evaluating Service-Learning, however, the 
consistent subtitle is “Telling the Story that Matters.” 

Mintz and Hesser (1996) have proposed a practice- to-principles-to-practice model for 
service-learning programs. They have identified five sets of principles, each developed within a 
specific context of practice, which have guided the practice of service-learning. Based on their 
own participation in service-learning and learning from dialogue which produced these five sets 
of principles during the past three decades, Mintz and Hesser identify three meta-principles — 
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collaboration, reciprocity, and diversity — and suggest they are useful lenses of the service-learning 
kaleidoscope for assessing the interactions of principles and practice. Adopting their model and 
its rationale provides guidelines in our search for meaningful data. 

Collaboration 

If the purpose of collaboration is to create a shared vision and joint strategies, then each 
partner must be heard, acknowledged, and addressed. Applying the lens of collaboration to the 
task of data gathering means that all partners (a) have an acknowledged voice in deciding what 
to collect and how to collect it, (b) assist with collecting data, and (c) learn from the data being 
studied. 

Reciprocity 

“Reciprocity suggests that as students learn from teachers, the community, and one another, 
teachers should also learn from students, the community, and one another, and so on” (Mintz & 
Hesser, 1996, p. 36). Applying this concept to data gathering, students will be gathering data 
from teachers, other students, and the community while teachers are gathering data from 
students, other teachers, and the community and so on. 

Diversity 

Service-learning involves a wide range of differences including racial, ethnic, gender, socio- 
economic status, ages, geographic affinities, sexual orientations, and physical and mental abilities. 
“When service-learning is viewed through the lens of diversity, opportunities... to discover gifts 
and capacities of all of the partners are increased. Diversity is an asset that brings to the process 
varied resources, talents, knowledge, and skills.” (Mintz & Hesser,. 1996, p. 38). This lens is 
essential to gathering meaningful data that will help tell the stories that matters. 

Sample Evaluation Plan 

Samples of questions that Mintz and Hesser (1996) have identified for each lens are 
structured on the chart on the next pages. For each question, suggestions are provided for the 
type of data that could be gathered to answer the question. 
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Evaluation Question 


Documentation Needed 


1) Have people been engaged in 

responsible and challenging actions for 
the common goal? 


List of initial participants 

Job descriptions, list of volunteers, and 
assignments 


2) Are teachers, community members, and 
students involved in the critical reflections? 


Examples of reflections by teachers, 
community members, and students 

Analysis of reflection according to identified 
themes of interest 


3) Are there opportunities for shared 
reflection among partners? 


List of dates and descriptions of opportunities 
for shared reflection 




List of new understandings gained or outcomes 
of shared reflections 


4) Are there clear service and learning goals 
for everyone involved? 


List of service and learning goals 


5) Do the learning and service goals 
encourage the enhancement of the 
capacities of all involved? 


Analysis or written rationale of how the learning 
and service goals encourage the enhancement of 
the capacities of all involved 




Examples of increased capacities from each 
group of partners 


6) Do all partners have the opportunity 
to define their needs? 


Copies of documents that summarize the needs 
identified by each group of partners 


7) How does attention to reciprocity shift 
the focus from needs and deficiencies to 
needs and capacities? 


Copies of asset mapping outcomes 

O 4 
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EVALUATION MATRIX 



Evaluation Question 


Documentation Needed 


8) Does the program clarify the responsibilities 
of each person and organization involved? 


Job descriptions, list of volunteers and 
assignments, copies of agreements 


9) Are changing circumstances of service 
providers and service needs recognized? 


Description of the system of checks & balances 
that is in place 

Examples of how the system has worked for 
every partner group 


10) Are the partners giving each other access to 
appropriate resources that augment their 
capacities to learn and serve one another well 
into the future? 


List of resources that have been shared 


11) Are training, supervision, monitoring, 
support, recognition, and evaluation 
designed to accommodate diverse 
backgrounds, orientations, and styles? 


Copies of training materials 


12) Does the program encourage participation 
in events for individuals with a variety 
of schedules? 


Log of dates and times of scheduled events 


13) Does the program extend to persons of 
various ages, socioeconomic levels, sexual 
orientation, physical and mental abilities, as 
well as race and gender? 


Description of strategies implemented to 
extend the program to persons of each 
identified characteristic 


14) Is participants’ involvement in various 
components of the program limited by 
which domain they represent or by the 
viewpoints they express? 


Survey results of participant perception 
regarding limitations to full participation 
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Reflection Framework 

Collecting the information that truly delivers the meaning of service-learning to the 
participants can be gathered through reflection activities with all participants in the service- 
learning experience. The National Youth Leadership Council has developed a very useful 
framework for reflection. Responding to the questions ‘What?^ So What?^ and Now What? 
provides students with a rather simple way to structure their reflective thinking. Written 
reflections provide an especially rich source of data for understanding meaning. Reflection 
products can be analyzed using a variety of categories. For example, if interested in the impact 
that participation in service-learning has on the development of complex thinking skills, one 
might collect written student reflections and apply the structure of analysis based on Blooms 
Taxonomy illustrated below. 



Question 


Process 


Outcome 


What? 


Recall 

Basic Description 


Learning: Knowledge (Recall) 
Experience: Preparation, Feelings 


So What? 


Analysis & Synthesis 


New Understandings 

Generalizations 

Eliminating Stereotypes 

Meaning 

New Meaning 

Self Understanding 

(Why you feel the way you do) 


Now What? 


Evaluation/Judgment 

Change 


New Practices 

New Attitudes, Beliefs, Values 
Improved Practice 
Renewed, Deepened or 
New Commitment 
Creative Problem Solving 



These are examples of some of the many tools you can use to evaluate service-learning. 
As you collect your data, be sure the 'meaning making ’ has a chance to come through. 
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DATA ANALYSIS 

Here's where the fun really begins! The results start to emerge and become clear. 
If there is an impact the data analysis stage is where it will be revealed. 



DATA ANALYSIS IS FUN AND EASY 



by Susan Root 

Various tools for data analysis are available to action researchers. Data analysis is a key step in 
action research, because the results of analysis form the basis for teachers conclusions about their 
classrooms and their decisions about new teaching techniques. 

When most teachers think of data analysis, they envision the quantitative (statistical) techniques 
used in formal educational studies. In formal research, the goal is to generalize from a sample to a 
larger population. The goal of action research, however, is to gain an in-depth understanding of a 
single situation. The teacher wants to know, for example, “How did mentoring affect my students’ 
academic performance?” or “Will initiating a recycling program enhance our students understanding 
of environmental concepts?” In this type of analysis, qualitative tools predominate. 

The first step in qualitative data analysis is data reduction. Action research projects typically 
generate a wealth of data, e.g., logs, interview transcripts, and student journals. Data reduction is the 
process of simplifying and assigning interpretive labels to this data. Although a variety of methods 
exist, two of the most widely used data reduction techniques are coding and thematic analysis. 
Coding is the process of assigning category labels to small units of data, such as phrases or 
paragraphs. The goal of coding is to allow the teacher to later pull out all examples from the data 
which pertain to a particular issue. For example, a teacher may wish to examine the impact of an 
intergenerational writing project on students’ attitudes toward senior citizens. He might assign the 
label SEN to each sentence in which students express feelings about the elderly. The teacher might 
refine this coding scheme further to record whether the feelings expressed are positive (SEN-POS), 
negative (SEN-NEG), or neutral (SEN-NEU). 

Thematic analysis, like coding, involves categorizing raw data. However, in thematic analysis, the 
researcher searches for patterns or themes in large amounts of data. For example, a teacher whose 
class is investigating a recent chemical spill may discover that, in their journals, students increasingly 
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write about contacting elected officials or circulating petitions about the problem. Thus, the teacher 
may conclude that a major theme in the data is increased knowledge of the political process. 

Once researchers have simplified the data by assigning codes and themes, the next step is to 
organize the results by summarizing the data in order to draw conclusions. To organize the results, 
the researcher identifies higher order relationships such as frequencies of occurrence or possible cause 
and effect connections among events. One way to organize results is to use a data display, such as a 
table or figure. For example, imagine a teacher who is interested in the effects of a service-learning 
project on the participation of high-, average-, and low-achieving students in her class. After coding 
indicators of participation in her field notes, such as asking questions, the teacher might create a 
table with participation on the vertical axis and each of the three groups listed along the horizontal 
axis, making a graphic comparison of participation levels in the project. 

A second type of data display is a correlative network. A correlative network is a graphic 
representation of the independent and dependent variables in a study and the possible cause-effect 
relations among them. For example, a teacher might be interested in the effects of three instructional 
techniques: a video, a cooperative learning task, and a clean-up project on students understanding of 
the problem of waste. The teacher might code students journals after each experience for concepts 
about waste and then create a correlative network representing the links between type of learning 
experience and type and number of concepts recorded by students. 

After teachers have organized the results of their research, the final step in data analysis is drawing 
and validating conclusions. Drawing conclusions is the process of creating an overarching construct 
which integrates the data from a project. For example, a teacher who studied the effects of a project 
in which students helped senior citizens write poetry might have uncovered a key finding: that the 
seniors were more interested in being able to recount their experiences than in learning the 
conventions of poetry. 

Because action researchers typically work alone as participants in the setting under study, their 
conclusions can be influenced by several types of errors. For example, a teacher may selectively 
attend only to evidence which confirms her hypotheses or may overemphasize the importance of a 
single event that occurred during the study. If they are to insure that conclusions are valid, action 
researchers must employ safeguards for reducing potential sources of error. The most widely used 
method for checking the validity of ones conclusions is triangulation. In triangulation, the researcher 
obtains evidence pertaining to a question from several sources. For example, a teacher may believe 
that the class mentoring program has positively affected student self-esteem. In order to confirm this 
finding, the teacher should seek evidence of increased self-esteem not only in students journals, for 
example, but in interviews and in surveys of students’ parents. 
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FINDINGS AND FINAL REPORTS 

As you collect your findings, remember that although formal reports, necessary fior funders, 
may tell the whole story, many of the audiences you are trying to reach probably will not 
take the time to read lengthy documents. There are more appropriate ways to present your 
research and evaluation data. 



WRITING WITH THE AUDIENCE IN MIND 



by Carl I. Fertman 

This section discusses how to report what we learn from an evaluation. It is important to focus 
on the way in which evaluation findings are presented. The packaging and timing of the reports are 
critical in increasing the impact of the evaluation. Those targeted by the evaluation — the teachers, 
principals, students, parents, funders, service recipients, and community members — are constantly 
receiving written information, and competition for their time and energy is keen. Reports must be 
presented in a way that will hold their attention long enough for the information to be conveyed. 

Writing With a Specific Audience in Mind 

Begin by identifying the people to whom you want to report the results of your evaluation. 
There are two groups to consider: those who would most benefit from the evaluation information, 
and those whose support is most needed. With an audience identified, the report should focus on 
information that will be of greatest interest to that audience. 

Principals, community leaders, advisory board members, and teachers (those already involved) 
need information on how service-learning is functioning. They can use that information to 
determine ways in which to strengthen the schools service-culture and commitment to service- 
learning. Colleagues and community supporters, on the other hand, will use the information to 
decide whether or not to become involved. With this audience, the goal is to answer questions and 
concerns so that they feel comfortable entering collaborations and offering resources. In this case, 
the information should be non- threatening, to overcome the anxieties and fears of the audience. 
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Resource brokers and opinion shapers (e.g., school board members, elected governmental 
officials, and community leaders) need information that will persuade them to consider changes in 
policy, resource allocation, and program activity. Community members in general are looking for 
information on what has already been accomplished. They want to hear about tangible outcomes, 
particularly those that show the impact of service-learning on the community. Community-based 
organizations that have already established service-learning collaborations will want to know about 
participant outcomes, staff effectiveness, and potential obstacles to implementation. Think about 
the members of your audience and the information that will be of greatest interest to them, and 
then tailor your report to their needs. 

General Recommendations 

These are a few points to keep in mind when you begin to write your evaluation report. 

Start with a plan. 

Think about the information above, and decide which audience and format are most 
appropriate for your evaluation. You may also want to tailor your presentation to a specific group 
or individual. Remember to plan ahead; don’t wait until you have your results to think about how 
you will share them. 

Keep it simple. 

Evaluation reports do not need to be elaborate. It is most important that the information shared 
is clear, simple, and timely. Use brief sections and subsections, and make titles clear and 
informative. Whenever possible depict findings pictorially, in charts, graphs or figures, and 
combine these with explanations in the text. Mix didactic and data-rich information with 
supporting evidence and anecdotal descriptions in the evaluation. This will make the report more 
interesting and readable and the findings more believable. 

Respect adult learning styles. 

There are three principles of adult learning to keep in mind when communicating evaluation 
findings. First, adults are most interested in information that is directly relevant to the projects and 
problems they are dealing with in their own lives. Second, they are most likely to use information 
that relates to their own personal experience. Third, different people learn in different ways — some 
are visually-oriented, others prefer narrative text, and some learn best when they hear something 
instead of reading it. Thus, it may be most beneficial to combine a few different methods of 
information dissemination. 
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COMMUNiaXION 
AND DISSEMINATION 

Dissemination of your results is the final stage of the research and evaluation process. 
Without this step, all your hard work will be for naught. This is the time to utilize your 
creativity as you strive to reach your dififrent audiences. 



SHARING WHAT WE LEARN 



by Carl L Fertman 

Sharing evaluation results is critical; if an evaluations findings are not distributed, then the 
purpose of conducting that evaluation is nullified. Evaluations are meant to provide information 
that will result in wider recognition and positive change. Without the report itself, evaluation 
information goes no further than the evaluators. It is our responsibility to let others know what 
we have learned. 

Packaging Reports to Compete for Attention 

In this information-rich society, there is much competition for your audiences attention. 
Keep in mind that there are a number of ways to disseminate evaluation information. Too many 
evaluators think that the only option is to write a technical evaluation report. While this may be 
appropriate in many situations, there are other alternatives, some of which may be more 
desirable depending on the audience and circumstances in question. 

Technical Report — This complete report of the evaluation includes background, methods, 
results, findings, and recommendations. It is a good choice to present to funders. 

Executive Summary — This report highlights evaluation results. It is appropriate for an audience 
of principals and administrators. 
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Highlight or Brief Paper — ^An overview of the evaluation that emphasizes recommendations and 
implications, providing a quick snapshot of evaluation results. 

Popular Article — ^This report will discuss the results and implications for a specific target group 
of teachers, parents, or principals. An article may be 'especially useful in reaching local and state 
teachers’ and principals’ associations. 

News Release or Press Conference — These communication methods will use the media for 
general distribution of information to the community via radio, television, and newspaper. 

Public Meeting — ^An information session designed to reach the public. It is good for 
communicating with parents, and can be presented as part of parents night. 

Media Appearance — ^This communication method presents evaluation information as part of a 
television or radio program, often on a public access channel through a local cable company. 
Media coverage conveys a sense of importance about a program, increasing the possibility that 
evaluation results will be heard and taken seriously. 



Include Your Students and Community Partners — Get these folks involved in the dissemination 
process. They, too, can and should be involved in getting the word out to the public. 

Staff Workshop — This method will use evaluation information as part of an in-service training 
for teachers and agency staff. 

Clearinghouse Submissions — The National Service-Learning and ERIC Clearinghouses have 
interest in teacher evaluations and service-learning. Submit copies of technical reports or 
highlight papers to these clearinghouses. 

Brochure, Poster, or Flyer — These communication methods will present findings in a concise 
and attractive manner. It is good for distributing information at conferences or through the mail. 

School Web Page — A homepage on the Internet can include general information about the 
school as well as specific service-learning evaluation information. A Web page has the advantage 
that it can be updated as often as desired. It can be combined with other methods listed here 
(newsletter, brochure, memo, etc.) to let your audience know that the homepage exists. 
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Memo — brief memo concerning the evaluation can be sent to select individuals and 
stakeholders. 

Personal Discussion — This communication method includes an individual or small group 
meeting to discuss the results and implications of the evaluation. 

Newsletter— Many schools already print newsletters. These often include such items as current 
service activities, recent accomplishments, kudos, personal stories, details of upcoming events of 
interest, and future directions for programs. Evaluation results can also be posted in a schools 
newsletter. 



A Final Word 

Once you have some data that show the value of your service-learning program, let your 
dissemination of these findings be a continuous part of your role as a service-learning teacher. 
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A FURTHER SUGGESTION 

Involving students is not innovative in service-learning; it is, however, an innovative and 
effective approach for research and evaluation. This section makes the case for student 
involvement in the process. 



INVOLVING STUDENTS IN EVALUATION 



by Robert D. Shumer 

Service-learning requires students to engage in meaningful service. Embedded in this process 
is a need to determine what service should be provided, what learning is needed, and what effect 
the whole process has on students, communities, and school organizations. Yet most of these 
issues cannot be addressed unless students, themselves, engage in collecting data and information 
as a normal part of their service-learning activities. 

Students have several opportunities to engage in evaluation while they perform work in high 
quality service-learning programs. First, they can help determine actual community needs and 
assets. Surveying and interviewing community members and evaluating the data collected can set 
the agenda for the service activities. Second, students can evaluate the implementation plan and 
activity effect by maintaining journals, oral records, and documents which describe the impact of 
the service on the recipients and the students. If, for example, students are engaged in cross-aged 
tutoring, records can be kept of the tutees progress in terms of number of books read, 
improvement in vocabulary, and even pre- and post-reading grade level tests. All this data can be 
used by students to assess the effectiveness of their service. The students can also maintain a 
written record of their own progress, describing what is learned about younger children, the 
reading process, and their own language arts skills. 

In another approach, students can actually participate as evaluators. As described by 
Campbell, Edgar, and Halsted (1994), students can do evaluations as their service activity. They 
can interview and observe recipients of service and describe the effect from the perspective of an 
“outside” person. By engaging in external evaluation processes, students can enhance their oral 
communication skills, as well as their analytical and critical thinking. The information they 
provide back to organizations or programs is a great form of service! 
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SUMMARY 



MAKE THE CASE FOR SERVICE-LEARNING 

by Marty Duckenfield 

The group of authors in this guidebook originally shared their expertise with teachers just 
like you at the 1997 National Service-Learning Conference in Orlando, Florida. In that full-day 
workshop, teachers had the opportunity to design a research and evaluation plan for their 
service-learning programs. Now you have the opportunity to do the same! 

As a new researcher in the field of service-learning, you have an exciting opportunity to 
advance the field. You can be a part of a vital and necessary effort to provide a wide variety of 
indicators that can show how service-learning is meeting the educational needs of all our 
students. 

The collective evaluation of teachers and their students from all across the country can 
expand the knowledge and understanding of this promising teaching method. By providing data 
acquired from the frontlines — looking at issues that reflect academic outcomes, citizenship, social 
and personal growth, workplace skills, teacher re-engagement, as well as community 
development — you can provide some real information that can guide the careful growth of this 
movement. 

The opportunity to be involved in the research and evaluation process will not only provide 
the service-learning field with the answers to so many questions; it will also provide you with a 
better service-learning program. As you enhance your skills as an evaluator, and if you involve 
your students as well, you will have a wealth of information to reflect on as you improve your 
own practice of service-learning. 

Come join us; make the case for service-learning so service-learning can continue to thrive. 
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ABOUT NYLC... 



Founded in 1983, National Youth Leadership Councils mission is to engage young people in 
communities and schools through innovation in learning, service, leadership and public policy. 
NYLC has been a pioneer in the field of service-learning — z method of teaching that enriches 
learning by engaging students in meaningful service to their schools and their commumties. 
Through careful integration with established curricula, lessons gained from hands-on service 
heighten interest and enhance academic achievement. Service-learning is a proven key to 
educational reform that also makes significant conuibutions to community development. 

For membership, publications or program information, contact: 

National Youth Leadership Council 
1910 West County Rd. B 
St. Paul, MN 55113 
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Phone: (612) 631-3672 

Fax: (612) 631-2955 

email: nylcusa@aol.com 

web site: http://www.nylc.org 
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